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Abstract 

This study assessed the relationship between students’ ability to interpret and to 
apply research methodology and their critical thinking skills. Participants were 103 graduate 
students from various disciplines, enrolled in six sections of an introductory-level educational 
research course at a southeastern university. These students were administered the 
California Critical Thinking Skills Test (CCTST), a multiple-choice test that targets core 
critical thinking skills regarded to be essential elements in a college education. Conceptual 
knowledge, which involved students' knowledge of research concepts, methodologies, and 
applications, was measured via comprehensive written midterm and final examinations. 

Findings revealed moderate statistically significant relationships between overall 
critical thinking skills and the midterm (r = .34, p < .001) and final (r = .26, p < .01) 
examination scores. Additionally, canonical correlation analyses indicated that both 
achievement scores were related to analysis, evaluation, and inference CCTST subscales 
(R c = .35, p < .05). 
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Relationship Between Critical Thinking and Performance in Research Methodology Courses 

According to the American Philosophical Association (1990), critical thinking is the 
process of purposeful, self-regulated judgement. Moreover, this construct is the cognitive 
engine which promotes problem-solving and decision-making. Central to critical thinking are 
the cognitive skills of interpretation, analysis, evaluation, inference, and explanation. As 
noted by Facione, Facione, Blohm, Howard, and Giancario (1998), these cognitive skills are 
utilized interactively in the reflective reasoning process that involves formulating a decision 
and solving a problem. In particular, while engaged in this process, an individual not only 
attempts to determine in a selective manner what to believe and what actions to take, but 
the person also possesses the capacity to apply appropriately the core critical thinking skills 
(Facione et al., 1998). 

As noted by Facione et al. (1998), people who possess critical thinking skills are apt 
to provide reasoned consideration to context, evidence, theories, procedures, and criteria 
in order to form a purposeful judgement. Thus, an important goal of higher education is to 
promote and to foster critical thinking skills (McBride & Reed, 1998). In particular, the 
National Institute of Education (1984) recommended that university-level curricula provide 
for “the development of capacities of analysis, problem solving, communication, and 
synthesis” (p. 43). Similarly, the Association of American Colleges (1985) advocated strongly 
that students learn inquiry skills, critical analytical skills, abstract logical thinking skills, and 
the like. Indeed, many of these skills have been identified as important in interpreting and 
applying research methodology at the graduate level (Onwuegbuzie, 1997). For example, 
a common skill that is emphasized by instructors of research methodology courses is the 
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ability to evaluate published research articles in their fields utilizing principles of the scientific 
method (Wilson & Onwuegbuzie, 1999). 

As conceptualized by Burns and Grove (1987), critiquing a research article involves 
at least five stages. The first stage involves identifying the study’s elements, as well as 
understanding the nature, significance, and meaning of both implicit and explicit 
components. This stage could be categorized as the analytical component of critical 
thinking. The second step of article critiques necessitates the interpretation of the 
nomenclatures, philosophies, theories, and concepts utilized in the report in a manner 
consistent with the researcher(s). This phase is consistent with the interpretational aspect 
of critical thinking. With respect to the third stage, it is essential that students have extensive 
knowledge of what each step of the research process comprises in order to assess the 
extent to which the article follows this process. This level is compatible with the evaluational 
skills associated with critical thinking. In the fourth step, it is necessary that students are able 
to identify the expressed and unexpressed assumptions of the researcher, as well as to 
examine the theoretical aspects of the study. Further, students must be aware of the links 
between the elements of the study, as well as relationships between components of the 
study and extant research. The explanation component of critical thinking is the hallmark of 
this stage. Finally, conceptual clustering (Werley & Fitzpatrick, 1985) must be undertaken, 
which maximizes the meaning attached to research findings, highlights gaps in the 
knowledge base, and generates new research questions. This stage is consistent primarily 
with the inferential aspect of critical thinking, as well as with the interpretational, analytical, 
evaluational, and expository stages. Thus, each of these five stages of the article critique 
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process appears to utilize one or more components of critical thinking. 

The current assessment reform movement in research methodology and statistics 
courses encourages instructors to think more broadly about cognitive measures which 
assess student learning (Garfield, 1994; Lesh & Lamon, 1992; Romberg, 1992). In 
response, instructors of these classes have begun incorporating innovative methods of 
assessment into their courses, one of the most common of these being authentic 
assessment (Onwuegbuzie, 2000). Authentic assessment is a method of collecting 
information concerning students’ understanding in contexts which reflect real-life situations, 
and which challenge students to apply what they have learned in their courses in authentic 
settings (Archbald & Newmann, 1988). In an attempt to incorporate authentic assessment 
into their courses, an increasing number of research methodology instructors are assigning 
their students research projects, whereby students conduct action research studies, in which 
real data are collected, analyzed, and interpreted. 

According to Schwartz, Slate, and Onwuegbuzie (1999), action research involves an 
eight-step cyclical process, as follows: (a) identifying an issue or problem to investigate; (b) 
gathering and reviewing relevant literature; (c) formulating research questions and/or 
hypotheses; (d) developing a research plan of action; (e) implementing the research plan; 
(f) analyzing the data and interpreting the findings; (g) communicating the findings; and (h) 
repeating the cycle with a modified problem or strategy derived from what was learned in the 
previous cycle (i.e., problem redefinition), until the research question is addressed in its 
entirety. As is the case for article critiques, these eight phases each appear to involve at 
least one of the cognitive skills associated with critical thinking (i.e., interpretation, analysis, 
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inference, evaluation, and explanation). 

Interestingly, several researchers have reported a significant relationship between 
critical thinking skills and disposition and various indices of academic aptitude and 
performance. For example, using the Watson-Glaser Critical Thinking Appraisal (Watson & 
Glaser, 1980), Behrens (1996) found that scores on the critical thinking test was a better 
predictor of nursing education performance, as measured by grade point average, than was 
prior high school grade point average or class ranking. Similarly, the CCTST has been found 
to be positively associated with scores on the verbal and mathematics portions of the 
Scholastic Achievement Test and on the ACT, with correlations ranging from .40 to .55 
(Facione et al., 1998). Also, Facione et al. (1998) reported a significant relationship between 
CCTST scores and college grade point average, as well as scores on the Nelson-Denny 
Reading Test (Brown, Fishco, & Hanna, 1993). At the graduate level, the CCTST has been 
found to be significantly related to scores on the verbal, quantitative, and analytic portions 
of the Graduate Record Examination (GRE), as well as to GRE total scores, with 
correlations ranging from .58 to .72. 

However, there is limited empirical evidence assessing the degree that critical 
thinking skills influence student performance in research methodology courses. Thus, the 
purpose of the present investigation was to assess the relationship between students’ 
achievement in research methodology courses, as measured by their ability to understand 
the research process, and their critical thinking skills. It was hypothesized that students who 
demonstrate the highest critical thinking skills also would attain the highest level of overall 
performance in research methodology courses. 
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Method 

Participants 

Participants were 103 graduate students from various disciplines, enrolled in six 
sections of an introductory-level educational research course at a southeastern university 
The same instructor taught all sections of the course. Of the sample, 88.3% were female. 
The proportion of female students in these sections was typical of the proportion of females 
enrolled in graduate programs in the College of Education at the time in which the study took 
place. All participants were enrolled in master’s degree programs. The ages of the 
participants ranged from 22 to 51 ( M = 28.3, SD = 7.5). . 

Instruments and Procedure 

Students were administered the California Critical Thinking Skills Test (CCTST; 
Facione, 1990a, 1990d, 1992) on the first day of class before any lecturing had begun. The 
CCTST is a 34-item, 4-option multiple-choice test that targets core critical thinking skills 
regarded to be essential elements in a college education. Moreover, the CCTST has been 
used effectively with graduate and professional school students (Facione et al., 1998). High 
scores on the scale (total = 34) indicates high critical thinking skills. The length of time 
necessary to administer the CCTST is 45 minutes. 

The CCTST was developed following a 46-member “American Philosophical 
Association’s Delphi consensus conceptualization of critical thinking (American Philosophical 
Association [APA], 1990). This instrument yields six scores: an overall score of one’s critical 
thinking skills and five subscale scores. The five subscales are analysis, evaluation, 
inference, deductive reasoning, and inductive reasoning. The first three CCTST subscales 
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(i.e., analysis, evaluation, inference) encapsulate the major core skills identified in the theory 
of critical thinking advanced in the Delphi Report (APA, 1990). These were the three 
subscales that were used in the present investigation. 

Analysis includes the sub-skills of categorization, clarifying meaning, examining ideas 
and identifying and analyzing arguments. Interpretation represents assessing claims and 
contentions, querying evidence, conjecturing alternatives, and drawing conclusions. 
Evaluation includes the sub-skills of stating results, justifying procedures, and presenting 
arguments. Each of the 34 items on the CCTST represents one of these three subscales. 
Interestingly, the CCTST is considered to be the best commercially-available instrument 
measuring critical thinking skills (Facione, 1990a). Scores from the 1989-1990 validation 
study of the CCTST yielded classical theory alpha reliability coefficients ranging from .68 to 
.70 (Facione, 1990c). For the current inquiry, scores pertaining to the CCTST had a 
classical theory alpha reliability coefficient of .72. 

Conceptual knowledge, which involved students’ knowledge of research concepts, 
methodologies, and applications, was measured via comprehensive written midterm and 
final examinations. These examinations consisted of open-ended questions, involving items 
that required knowledge of the research process. All of the items on the midterm 
examination form pertained to content from the first half of the course and were chosen from 
the instructor's item bank to ensure that the examination was typical of past examinations 
given by the instructor. The final examination also was developed by the course instructor 
and paralleled the format of the midterm examination, yet covered the complete course 
content. Both the midterm and the final examination were administered under untimed 
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conditions and were scored on a 100-point scale by the instructor, using a key that specified 
the number of points awarded for both correct and partial-credit answers. 

Results 

The means and standard deviations pertaining to the full CCTST scale and the 
analysis, evaluation, and inference subscales are presented in Table 1. These means were 
compared to the means reported by the developers of the CCTST (Facione, 1990b). The 
graduate normative group used in Facione’s (1990b) study involved 153 nursing students 
working toward their Master’s degrees in a nationally-ranked nursing program in an urban 
community. Interestingly, the mean CCTST score reported by the graduate students in the 
present study (i.e., 16.21) was statistically significantly lower (f = -4.76, p < .0001) than that 
computed for participants in Facione’s (1990b) inquiry ( M = 19.01, SD = 5.09). The effect 
size associated with this difference was moderate to large. Thus, the graduate students in 
the current study had a lower propensity for critical thinking skills than did the induction 
sample. Additionally, a median percentile rank equivalent score (MPRES) was calculated 
by comparing the median score for the full CCTST scale in the present study to the 
percentile rank norms reported by Facione (1990b) for the graduate nursing students. The 
calculated MPRES of 32 for the full CCTST scale indicates that at least 50% of the present 
sample scored higher than did 32% of the normative group. Thus, it is clear that the 
participants in the current investigation represented a cohort of students who had low overall 
critical-thinking scores. 
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Insert Table 1 about here 



The Shapiro-Wilk test (Shapiro & Wilk, 1965; Shapiro, Wilk, & Chen, 1968) did not 
indicate that the distribution of overall CCTST scores (W = .99, p > .05) was non-normal, 
Similarly, the distribution of the midterm {W = .98, p > .05) and final (W = .99, p > .05) 
examination scores did not appear to depart form normality. Thus, the use of correlational 
analyses appeared to be justified. Pearson’s product-moment correlation coefficients, using 
the Bonferroni adjustment, revealed moderate statistically significant relationships between 
overall critical thinking skills and the midterm (r= .34, p < .001) and final (r= .26, p < .01) 
examination scores. Specifically, students with the highest scores on the CCTST tended 
also to attain the highest achievement scores in the research methodology class at both the 
midpoint and at the end of the course. Table 2 presents the correlation matrix involving the 
three CCTST subscales and the two achievement measures from which the canonical 
correlation analysis was undertaken. Because the purpose of the study was to examine the 
canonical correlation model assumed to underlie these correlations, the correlation matrix 
was not interpreted. 



Insert Table 2 about here 



The strength of the relationship between the two sets of variables was assessed by 
examining the magnitude of the canonical correlation coefficients. These coefficients 
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represented the degree of relationship between the weighted dimension variables and the 
weighted achievement variables. In addition, the significance of the canonical roots was 
tested via the F-statistic based on Rao's approximation (Rao, 1952). 

The canonical analysis revealed that both canonical correlations combined were 
statistically significant (F [6 , 188] = 2.18, p < .05). However, when the first canonical root 
was excluded, the remaining canonical root was not statistically significant. Together, these 
results suggest that the first canonical function was statistically significant, but the second 
canonical root was not statistically significant. However, because the calculated probabilities 
are sensitive to sample size, particular attention should be paid to the educational (practical) 
significance of the obtained results (Thompson, 1980). The educational significance of 
canonical correlations typically are assessed by examining their size (Thompson, 1980, 
1984, 1988, 1990). The canonical correlation indicates how much variance the sets of 
weighted original variables share with each other (Thompson, 1988). In the present study, 
the first canonical correlation (R c1 = .35) appeared to be moderately educationally significant, 
contributing 12.2% (i.e., R c1 2 ) to the shared variance. However, the second canonical 
correlation (R^ = .07) did not appear to be educationally significant. Consequently, only the 
first canonical correlation was interpreted. 

Data pertaining to the first canonical root are presented in Table 3. This table 
provides both standardized function coefficients and structure coefficients. An examination 
of the standardized canonical function coefficients revealed that, using a cutoff correlation 
of 0.3 recommended by Lambert and Durand (1975) as an acceptable minimum loading 
value, all three dimensions of the CCTST (i.e., analysis, evaluation, and inference) made 
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an important contribution to the achievement composite-with evaluation being slightly the 
major contributor. With respect to the achievement set, both the midterm and final 
examination scores made an important contribution to the composite set, with the midterm 
scores making by far the largest contribution. 



Insert Table 3 about here 



The structure coefficients (Table 3) similarly revealed that all three CCTST 
dimensions made important contributions to the first canonical variate. The square of the 
structure coefficient (Table 2) indicated that evaluation and inference made large 
contributions, explaining 61.8% and 51.3% of the variance, respectively. Analysis made a 
moderate contribution. With regard to the achievement cluster, both the midterm and final 
examination made noteworthy contributions, with the midterm scores making an extremely 
large contribution-explaining 93.1% of the variance. 

Discussion 

The purpose of the current study was to determine the relationship between 
academic achievement in a research method course and critical thinking skills. Interestingly, 
assuming that the CCTST generates reliable and valid scores that are indicative of critical 
thinking skills, students in the current investigation tended to demonstrate less adequate 
critical thinking skills than did the inductive sample of graduate nursing students. However, 
this result is not surprising, bearing in mind that the issue of critical thinking skills and 
disposition has been a major focus in nursing education and research (Facione, Facione, 
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& Sanchez, 1994). Indeed, as noted by Facione and Facione (1996), the conceptual 
definition of critical thinking has particular application to the nursing field. For example, the 
cognitive skills of interpretation, analysis, evaluation, inference, and explanation that 
permeate critical thinking skills and dispositions are necessary for making clinical decisions 
in the nursing context. Moreover, critical thinking in the field of nursing implies both 
knowledge-based and clinical judgment skills, which are both essential components of 
nursing (Delk, 1999). 

The major findings were (a) a moderate statistically significant relationship between 
overall critical thinking skills and both the midterm and final examination scores; and (b) both 
achievement scores simultaneously were related to the analysis, evaluation, and inference 
components of critical thinking. This result is consistent with the bulk of the literature that has 
reported a significant relationship between critical thinking skills and disposition and various 
measures of academic performance (Behrens, 1996; Facione et al., 1998). 

Unfortunately, due to the correlational design used, the causal nature of the 
relationship between critical thinking skills and research methodology achievement is not 
clear. For example, the univariate and multivariate relationships found may indicate that 
critical thinking skills tend to increase performance levels in research methodology courses. 
This causal direction may occur because critical thinking skills such as interpretation, 
analysis, evaluation, inference, and explanation also are important skills utilized when 
conducting article critiques and research studies, as well as when attempting to understand 
research concepts and methodologies. 

However, the fact that the CCTST was administered on the first day of class before 
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any lecturing begun, does not necessarily mean that any causal relationship between critical 
thinking skills and research methodology achievement is uni-directional, with the former 
being the antecedent, mediator, or cause. In fact, it is likely that the relationship between 
these two constructs is bi-directional. For example, as students improve their critical thinking 
skills, their ability to understand the research process increases, as manifested by tasks 
such as being able to read, to understand, to synthesize, to evaluate, and to apply research 
articles in their fields. At the same time, as these same students improve their research 
skills, their critical thinking skills further improve, which subsequently enhance their research 
competency, and so on, until both critical thinking skills and research skills are maximized. 

The possibility of a bi-directional relationship between these two variables should be 
the subject of future studies. In particular, researchers could replicate Maynard’s (1996) 
longitudinal inquiry. Maynard (1996) examined the development of critical thinking skills from 
the beginning nursing student to the experienced professional nurse, as well as the 
relationship of critical thinking ability to professional competence over time. Interestingly, 
Maynard (1996) utilized Benner’s (1984) stages of skill acquisition (i.e., novice, advanced 
beginner, competent, proficient, and expert) as the theoretical framework. Maynard found 
that critical thinking skills increase significantly as the nurse practiced in the profession. 
Additionally, critical thinking skills were related significantly to nurses’ professional 
competence. Thus, researchers in the future should (a) compare students with different 
levels of expertise in research, and (b) study the critical thinking abilities of students as they 
progress from their first research methodology course (i.e., novice researcher), to the 
completion of their second or third research course (i.e., advanced beginner), to the 
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completion of their doctoral degrees (i.e., competent researcher), to the publication of their 
first few research articles (i.e., proficient researcher), and to the publication of several large- 
scale studies (i.e., expert). Also of interest is to compare levels of critical thinking skills 
between researchers who predominantly or exclusively utilize quantitative research 
methodologies and qualitative researchers. Such investigations will increase our 
understanding of critical thinking, as it impacts teaching and learning in the context of 
research methodology courses. 
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Table 1 

Means, Standard Deviations, and Medians Pertaining to Scores on the CCTST Scale and 
Subscale Scores 



Measure 


M 


SD 


Median 


CCTST Total Score 


16.21 


3.79 


16 


Analysis 


4.42 


1.43 


5 


Evaluation 


5.83 


2.27 


6 


Inference 


5.97 


1.67 


6 
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Table 2 



Intercorrelations Among all CCTST Subscales and Achievement Measures 



Theme 


1 


2 


3 


4 


1 . Analysis 










2. Evaluation 


.18 








3. Inferences 


.08 


.37 






4. Midterm 


.20 


.26 


.24 




5. Final 


.12 


.23 


.22 


.62 



* statistically significant after the Bonferroni adjustment 
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Table 3 

Canonical Solution for First Function: Relationship Between Achievement Scores and 
Scores on Three Subscales of the California Critical Thinking Skills Tests 



Variable 


Standardized 

Coefficient 


Structure 

Coefficient 


Structure 2 


Critical Thinking Dimension: 








Analysis 


.421' 


.544' 


.296 


Evaluation 


.538' 


.786' 


.618 


Inference 


.486' 


.716' 


.513 


Achievement: 








Midterm examination 


.761' 


.965' 


.931 


Final examination 


.332' 


.802' 


.643 



'loadings with effect sizes larger than .3 (Lambert & Durand, 1975) 
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